# Generated text resources used in the paper.

This document contains details on accessing generated text resources used in
the paper. [Return to main page.](README.md)

### Normalized native-script Wikipedia text.

The roughly 340MB tar file
[text/text_normalized_native_script_corpora.tar](https://storage.googleapis.com/gresearch/context_aware_transliteration/text/text_normalized_native_script_corpora.tar)
includes all text normalized native script Wikipedia corpora used for language
modeling in the paper, consisting of text resources originating in the [Dakshina
dataset](https://github.com/google-research-datasets/dakshina).  The text
normalization involved mapping out-of-vocabulary characters to whitespace, as
detailed in the paper.

### Simulated parallel transliteration corpora.

Using the above native script Wikipedia, we simulated parallel transliterated corpora by automatically romanizing the text.  See paper for methods for sampling from possible romanizations. This was done with 5 differently trained models, to capture variability over multiple training runs. Links to download the data are provided in the following table:

File | Download | Size (GB) | Notes
---- | :------: | :--------: | :------
**text/text_simulated_parallel_corpora_a.tar** | [link](https://storage.googleapis.com/gresearch/context_aware_transliteration/text/text_simulated_parallel_corpora_a.tar) | 6.6 | Simulated parallel transliterated sentences generated by sampling automatic romanizations, run A.
**text/text_simulated_parallel_corpora_b.tar** | [link](https://storage.googleapis.com/gresearch/context_aware_transliteration/text/text_simulated_parallel_corpora_b.tar) | 6.6 | Simulated parallel transliterated sentences generated by sampling automatic romanizations, run B.
**text/text_simulated_parallel_corpora_c.tar** | [link](https://storage.googleapis.com/gresearch/context_aware_transliteration/text/text_simulated_parallel_corpora_c.tar) | 6.6 | Simulated parallel transliterated sentences generated by sampling automatic romanizations, run C.
**text/text_simulated_parallel_corpora_d.tar** | [link](https://storage.googleapis.com/gresearch/context_aware_transliteration/text/text_simulated_parallel_corpora_d.tar) | 6.6 | Simulated parallel transliterated sentences generated by sampling automatic romanizations, run D.
**text/text_simulated_parallel_corpora_e.tar** | [link](https://storage.googleapis.com/gresearch/context_aware_transliteration/text/text_simulated_parallel_corpora_e.tar) | 6.6 | Simulated parallel transliterated sentences generated by sampling automatic romanizations, run E.

### Off-line constructed transliteration caches.

To test the efficacy of on-line transliteration using cached transliterations from relatively expensive multi-system ensembles, we built transliteration caches for frequent Latin script tokens in each language.  See paper for methods for building the caches and sizes per language.  As with the simulated parallel corpora, cache building was done with 5 differently trained models, to capture variability over multiple training runs. Links to download the data are provided in the following table:

File | Download | Size (GB) | Notes
---- | :------: | :--------: | :------
**text/text_transliteration_caches_a.tar** | [link](https://storage.googleapis.com/gresearch/context_aware_transliteration/text/text_transliteration_caches_a.tar) | 2.8 | Generated transliteration caches for on-line processing, run A.
**text/text_transliteration_caches_b.tar** | [link](https://storage.googleapis.com/gresearch/context_aware_transliteration/text/text_transliteration_caches_b.tar) | 2.8 | Generated transliteration caches for on-line processing, run B.
**text/text_transliteration_caches_c.tar** | [link](https://storage.googleapis.com/gresearch/context_aware_transliteration/text/text_transliteration_caches_c.tar) | 2.8 | Generated transliteration caches for on-line processing, run C.
**text/text_transliteration_caches_d.tar** | [link](https://storage.googleapis.com/gresearch/context_aware_transliteration/text/text_transliteration_caches_d.tar) | 2.8 | Generated transliteration caches for on-line processing, run D.
**text/text_transliteration_caches_e.tar** | [link](https://storage.googleapis.com/gresearch/context_aware_transliteration/text/text_transliteration_caches_e.tar) | 2.8 | Generated transliteration caches for on-line processing, run E.

